Conversational speech analysis method, and conversational speech analyzer

ABSTRACT

The invention provides a conversational speech analyzer which analyzes whether utterances in a meeting are of interest or concern. Frames are calculated using sound signals obtained from a microphone and a sensor, sensor signals are cut out for each frame, and by calculating the correlation between sensor signals for each frame, an interest level which represents the concern of an audience regarding utterances is calculated, and the meeting is analyzed.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of application Ser. No.11/705,756, filed Feb. 14, 2007 now U.S. Pat. No. 8,036,898, thedisclosure of which is hereby incorporated by reference.

CLAIM OF PRIORITY

The present application claims priority from Japanese application JP2006-035904 filed on Feb. 14, 2006, the content of which is herebyincorporated by reference into this application.

FIELD OF THE INVENTION

The present invention relates to the visualization of the state of ameeting which, at a place where a large number of people discuss anissue. The interest level that the participants have in the discussionis analyzed, the activity of the participants at the meeting can beevaluated and the progress of the meeting can be evaluated for those notpresent at the meeting. By saving this information, it can be used forfuture log analysis.

BACKGROUND OF THE INVENTION

It is desirable to have a technique to record the details of a meeting,and many such conference recording methods have been proposed. Mostoften, the minutes of the meeting are recorded as text. However, in thiscase, only the decisions are recorded, and it is difficult to capturethe progress, emotion and vitality of the meeting which can only beappreciated by those present, such as the mood or the effect on otherparticipants. To record the mood of the meeting, the utterances of theparticipants can be recorded, but playback requires the same amount oftime as the meeting time, so this method is only partly used.

Another method has been reported wherein the relationships between theparticipants is displayed graphically. This is a technique whichdisplays personal interrelationships by analyzing electronicinformation, such as E-mails and web access logs, (for example, JP-A NO.108123/2001). However, the data used for displaying personalinterrelationships is only text, and these interrelationships cannot bedisplayed graphically from the utterances of the participants.

SUMMARY OF THE INVENTION

A meeting is an opportunity for lively discussion, and all participantsare expected to offer constructive opinions. However, if no livelydiscussion took place, there must have been some problems whose causeshould be identified.

In a meeting, it is usual to record only the decisions that were made.It is therefore difficult to fully comprehend the actions and activityof the participants, such as the topics in which they were interestedand by how much.

When we participate in a meeting, it is common for the participants toreact to important statements by some action such as nodding the head ortaking memos. To analyze the state of a meeting and a participant'sactivity, these actions must be detected by a sensor and analyzed.

The problem that has to be solved, therefore, is to appreciate how theparticipants behaved, together with their interest level and the moodand progress of the meeting by analyzing the information obtained frommicrophones and sensors, and by graphically displaying this obtainedinformation.

The essential features of the invention disclosed in the application forthe purpose of resolving the above problem, are as follows. Theconversational speech analysis method of the invention includes a soundcapture means for capturing sound from a microphone, a speech/nonspeechactivity detection means for cutting out speech frames and nonspeechframes from the captured sound, a frame-based speech analysis meanswhich performs analysis for each speech/nonspeech frame, a sensor signalcapture means for capturing a signal from a sensor, a sensor activitydetection means for cutting out the captured signal for each frame, aframe-based sensor analysis means for calculating features from a signalfor each frame, an interest level judging means for calculating aninterest level from the speech and sensor information for each frame,and an output means for displaying a graph from the interest level.

In this conversational speech analysis method, the state of a meetingand its participants can be visualized from the activity of theparticipants, and the progress, mood and vitality of the meeting, byanalyzing the data captured from the microphone and the sensor, anddisplaying this information graphically.

By acquiring information such as the progress, mood and vitality of themeeting, and displaying this information graphically, the meetingorganizer can extract useful elements therefrom. Moreover, not only themeeting organizer, but also the participants can obtain information asto how much they participated in the meeting.

The present invention assesses the level of involvement of theparticipants in a meeting and useful utterances in which a large numberof participants are interested. The present invention may therefore beused to prepare minutes of the meeting, or evaluate speakers who madeuseful comments by selecting only useful utterances. Furthermore, it canbe used for project management, which is a tool for managing a largenumber of people.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of a conversational speech analysis accordingto the invention;

FIG. 2 is an image of the conversational speech analysis according tothe invention;

FIG. 3 is a flow chart of the conversational speech analysis used in theinvention;

FIG. 4 is a speech/nonspeech activity detection processing andcorresponding flow chart;

FIG. 5 is a frame-based sound processing and corresponding flow chart;

FIG. 6 is a sensor activity detection processing and corresponding flowchart;

FIG. 7 is a frame-based sensor analysis process and corresponding flowchart;

FIG. 8 is an interest level judgment process and corresponding flowchart;

FIG. 9 is a display process and corresponding flow chart;

FIG. 10 is a speech database for storing frame-based sound information;

FIG. 11 is a database for storing frame-based sensor information;

FIG. 12 is an interest level database (sensor) for storing sensor-basedinterest levels;

FIG. 13 is an interest-level database (microphone) for storingmicrophone-based interest levels;

FIG. 14 is a customized value database for storing personalcharacteristics;

FIG. 15 is a database used for speaker recognition;

FIG. 16 is a database used for emotion recognition;

FIG. 17 is a time-based visualization of utterances by persons in themeeting; and

FIG. 18 is a time-based visualization of useful utterances in themeeting.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Some preferred embodiments of the invention will now be describedreferring to the drawings.

FIG. 1 is a schematic diagram of the invention for implementing aconversational voice analysis method. One example of the analyticalprocedure will now be described referring to FIG. 1. In order to makefor easier handling of data, ID (101-104) are assigned to microphonesand sensors. First, to calculate a speech utterance frame,speech/nonspeech activity detection is performed on sound 105 capturedby the microphone. As a result, a speech frame 106 is detected. Next,since a speech frame cannot be found from a sensor signal, activitydetection of the sensor signal is performed using a time T1 (107) whichis the beginning and a time T2 (108) which is the end of the speechframe 106. Feature extraction is performed respectively on the sensorsignal in a frame 109 and sound in the speech frame 106 found by thisprocessing, and features are calculated. The feature of the frame 109 is110, and the feature of the speech frame 106 is 111. This processing isperformed on all the frames. Next, an interest level is calculated fromthe calculated features. The interest level of the frame 109 is 112, andthe interest level of the speech frame 106 is 113. The calculatedinterest level is stored in an interest level database 114 in order tosave the interest level. Next, an analysis is performed using theinformation stored in the database, and a visualization is made of theresult. Plural databases are used, i.e., an interest level database 114which stores the interest level, a location database 115 which storesuser location information, and a name database 116 which stores thenames of participants. If the data required for visualization are aperson's name and interest level, an analysis can be performed by usingthese three databases. The visualization result is shown on a screen117. On the screen 117, to determine the names of persons present andtheir interest level, ID are acquired from the location of the interestlevel database 114 and location database 115, and names are acquiredfrom the ID of the location database 115 and ID of the name database116.

Next, the diagrams used to describe the present invention will bedescribed. FIG. 1 is a schematic diagram of conversational speechanalysis. FIG. 2 is a user image of conversational speech analysis. FIG.3 is a flow chart of conversational speech analysis. FIG. 4 is a flowchart of speech/nonspeech activity detection. FIG. 5 is a flow chart ofa frame-based speech analysis. FIG. 6 is a flow chart of sensor activitydetection. FIG. 7 is a flow chart of sensor analysis according to frame.FIG. 8 is a flow chart of interest level determination. FIG. 9 is a flowchart of a display. FIG. 10 is a speech database. FIG. 11 is a sensordatabase. FIG. 12 is an interest level database (sensor). FIG. 13 is aninterest level database (microphone). FIG. 14 is a customized valuedatabase. FIG. 15 is a speaker recognition database. FIG. 16 is anemotion recognition database. FIG. 17 is a visualization of the interestlevel of persons according to time in the meeting. FIG. 18 is avisualization of useful utterances according to time in the meeting.

According to the present invention, the interest level of participantsin a certain topic is found by analysis using microphone and sensorsignals. As a result of this analysis, the progress, mood and vitalityof the meeting become useful information for the meeting organizer. Thisuseful information is used as support to improve project administration.

An embodiment using the scheme shown in FIG. 1 will now be describedusing FIG. 2. FIG. 2 is an application image implemented by thisembodiment. FIG. 2 is a scene where a meeting takes place, many sensorsand microphones being deployed in the vicinity of a desk and theparticipants. A microphone 201 and sensor 211 are used to measure thestate of participants in real time. Further, this microphone 201 andsensor 211 are preferably deployed at locations where the participantsare not aware of them.

The microphone 201 is used to capture sound, and the captured sound isstored in a personal computer 231. The personal computer 231 has astorage unit for storing the captured sound and sensor signals, variousdatabases and software for processing this data, as well as a processingunit which performs processing, and a display unit which displaysprocessing analysis results. The microphone 201 is installed at thecenter of a conference table like the microphone 202 in order to recorda large amount of speech. Apart from locating the microphone in a placewhere it is directly visible, it may be located in a decorative plantlike the microphone 203, on a whiteboard used by the speaker like themicrophone 204, on a conference room wall like the microphone 205, or ina chair where a person is sitting like the microphone 206.

The sensor 211 is used to grasp of the movement of a person, signalsfrom the sensor 211 being sent to a base station 221 by radio. The basestation 221 receives the signal which has been sent from the 211, andthe received signal is stored by the personal computer 231. The sensor211 may be of various types, e.g. a load cell may be installed whichdetects the movement of a person by the pressure force on the floor likethe sensor 212, a chair weight sensor may be installed which detects abodyweight fluctuation like the sensor 213, an acceleration sensor maybe installed on clothes, spectacles or a name card which detects themovement of a person like the sensor 214, or a an acceleration sensormay be installed on a bracelet, ring or pen to detect the movement ofthe hand or arm like the sensor 215.

A chart which displays the results of analyzing the signals obtainedfrom the microphone 201 and sensor 211 by the personal computer 231 onthe screen of the personal computer 231, is shown by a conference viewer241.

The conference viewer 241 displays the current state of the meeting, anda person who was not present at the meeting can grasp the mood of themeeting by looking at this screen. Further, the conference viewer 241may be stored to be used for log analysis.

The conference viewer 241 is a diagram comprised of circles and lines,and shows the state of the meeting. The conference viewer 241 showswhether the participants at the meeting uttered any useful statements.The alphabetical characters A-E denote persons, circles around themdenote a useful utterance amount, and the lines joining the circlesdenote the person who spoke next. The larger the circle, the larger theuseful utterance amount is, and the thicker the line, the moreconversation occurred between the two persons it joins. Hence, bycomposing this screen, it is possible to grasp the state of theconference at a glance.

A procedure to analyze conversational speech will now be describedreferring to the flow chart of FIG. 3. In this analysis, it isdetermined to what extent the participants were interested in thepresent meeting by using the speech from the microphone 201 and thesignal from the sensor 211. Since it is possible to analyze the extentto which participants were interested in the topics raised at themeeting, it is possible to detect those utterances which were importantfor the meeting. Also, the contribution level of the participants at themeeting can be found from this information.

In this patent, an analysis is performed by finding a correlationbetween signals in speech and nonspeech frames. In the analyticalmethod, first, a frame analysis is performed on the speech recorded bythe microphone, and the frames are divided into speech frames andnonspeech frames. Next, this classification is applied to the sensorsignal recorded from the sensor, and a distinction is made betweenspeech and nonspeech signals. A correlation between speech and nonspeechsignals which is required to visualize the state of the persons present,is thus found.

Next, the conversational speech analysis procedure will be describedreferring to the flow chart of FIG. 3. A start 301 is the start of theconversational speech analysis. A speech/nonspeech activity detection302 is processing performed by the personal computer 231 which makes adistinction between speech and nonspeech captured by the microphone 201,and detects these frames. FIG. 4 shows the detailed processing.

A frame-based analysis 303 is processing performed by the personalcomputer 231 which performs analysis on the speech and nonspeech cut outby the speech/nonspeech activity detection 302. FIG. 5 shows thedetailed processing.

A sensor activity detection 304 is processing performed by the personalcomputer 231 which distinguishes sensor signals according to frame usingthe frame information of the speech/nonspeech activity detection 302.FIG. 6 shows the detailed processing.

A frame-based sensor analysis 305 is processing performed by thepersonal computer 231 which performs analysis on signals cut out by thesensor activity detection 304. FIG. 7 shows the detailed processing.

An interest level determination 306 is processing performed by thepersonal computer 231, which determines how much interest (i.e. theinterest level) the participants have in the conference, by usingframe-based information analyzed by the frame-based speech analysis 303and frame-based sensor analysis 305. FIG. 8 shows the detailedprocessing.

A display 307 is processing performed by the personal computer 231 whichprocesses the results of the interest level determination 306 intoinformation easily understood by the user, and one of the resultsthereof is shown graphically on the screen 241. FIG. 9 shows thedetailed processing. An end 308 is the end of the conversational speechanalysis.

The processing of the speech and/or nonspeech activity detection 302will now be described referring to the flow chart of FIG. 4. Thisprocessing, which is performed by the personal computer 231, makes aclassification into speech and nonspeech using the sound recorded by themicrophone 201, finds frames classified as speech or nonspeech, andstores them in the speech database (FIG. 10). A start 401 is the startof speech/nonspeech activity detection.

A speech capture 402 is processing performed by the personal computer231 which captures sound from the microphones 201. Also, assuming someinformation is specific to the microphone, it is desirable to store notonly speech but also information about the microphone ID number,preferably in a customized value database (FIG. 14) which manages data.

A speech/nonspeech activity detection 403 is processing performed by thepersonal computer 231 which classifies the sound captured by the speechcapture 402 into speech and nonspeech. This classification is performedby dividing the speech into short time intervals of about 10 ms,calculating the energy and zero cross number in this short timeinterval, and using these for the determination. This short timeinterval which is cut out is referred to as an analysis frame. Theenergy is the sum of the squares of the values in the analysis frame.The number of zero crosses is the number of times the origin is crossedin the analysis frame. Finally, a threshold value is preset todistinguish between speech and nonspeech, values exceeding the thresholdvalue being taken as speech, and values less than the threshold valuebeing taken as nonspeech.

Now, if a specific person recorded by the microphone is identified, aperformance improvement may be expected by using a threshold valuesuitable for that person. Specifically, it is preferable to use anenergy 1405 and zero cross 1406, which are threshold values for themicrophone ID in the customized value database (FIG. 14), as thresholdvalues. For the same reason in the case of, emotion recognition andspeaker recognition, if there are coefficients suitable for that person,it is preferable to store a customized value 1504 in the speakerrecognition database of FIG. 15 for speaker recognition, and acustomized value 1604 in the emotion recognition database of FIG. 16 foremotion recognition. This method is one example of the processingperformed to distinguish speech and nonspeech in sound, but any othercommon procedure may also be used.

A speech cutout 404 is processing performed by the personal computer 231to cut out speech from each utterance of one speaker. A speech/nonspeechactivity detection 403 performs speech/nonspeech detection, and thisdetection is performed in a short time interval of about 10 ms. Hence,it is analyzed whether the judgment result of a short time interval iscontinually the same, and the result is continually judged to be speech,this frame is regarded as an utterance.

To enhance the precision of the utterance frame, a judgment may be madealso according to the length of the detected frame. This is because oneutterance normally lasts several seconds or more, and frames less thanthis length are usually sounds which are not speech, such as noise.

This technique is an example of processing to distinguish speech fromnonspeech, but any other generally known method may be used. Further,when a frame is detected, it is preferable to calculate the start timeand the ending time of the frame. After cutting out both the speechframes and nonspeech frames based on the result of this activitydetection, they are stored in the memory of the personal computer 231.

A speech database substitution 405 is processing performed by thepersonal computer 231 to output frames detected by the speech cutout 404to the speech database (FIG. 10).

The recorded information is a frame starting time 1002 and closing time1003, a captured microphone ID 1004, a result 1005 of thespeech/nonspeech analysis, and a filename 1006 of the speech captured bythe speech cutout 404. The frame starting time 1002 and closing time1003 are the cutout time and date. Since plural microphones areconnected, the captured microphone ID 1004 is a number for identifyingthem. The result 1005 of the speech/nonspeech analysis is the resultidentified by the speech cut out 404, and the stored values are speechor nonspeech.

When the filename of the speech cut out by the speech cutout 404 isdecided, and speech is cut out from the result of the activity detectionby the speech cutout 404 and stored in the memory, the data is convertedto a file and stored. The filename 1006 which is stored is preferablyuniquely identified by the detected time so that it can be searchedeasily later, and is stored in the speech database. An end 406 is theend of speech/nonspeech activity detection.

The procedure of the frame-based sound analysis 303 will now bedescribed referring to the flow chart of FIG. 5. This processing isperformed by the personal computer 231, and analyzes the sound containedin this frame using the speech/nonspeech frames output by thespeech/nonspeech activity detection 302. A start 501 is the start of theframe-based sound analysis.

A sound database acquisition 502 is performed by the personal computer231, and acquires data from the speech database (FIG. 10) to obtainsound frame information.

A speech/nonspeech judgment 503 is performed by the personal computer231, and judges whether the frame in which the sound databaseacquisition 502 was performed is speech or nonspeech. This is becausethe items to be analyzed are different for speech and nonspeech. Bylooking up the speech/nonspeech 1005 from the speech database (FIG. 10),an emotion recognition/speaker recognition/environmental soundrecognition 504 is selected in the case of speech, and an end 506 isselected in the case of nonspeech.

The emotion recognition/speaker recognition 504 is performed by thepersonal computer 231 for items which are judged to be speech in thespeech/nonspeech determination 503. Emotion recognition and speakerrecognition are performed for the cutout frames.

Firstly, as the analysis method, the sound of this frame is cut intoshort time intervals of about 10 ms, and features are calculated forthis short time interval. In order to calculate the height (fundamentalfrequency) of the sound which is one feature. 1: The power spectrum iscalculated from a Fourier transform. 2: An auto correlation function isexecuted for this power spectrum. 3: The peak of the autocorrelationfunction is calculated. And, 4: The period of the peak is found, and thereciprocal of this period is calculated. In this way, the height(fundamental frequency) of the sound can be found from the sound. Thefundamental frequency can be found not only by this method, but also byany other commonly known method. The feature is not limited to theheight of the sound, and may additionally be a feature such as theinterval between sounds, long sounds, laughter, sound volume and soundrate, from which a feature for detecting the mood is detected, and takenas a feature for specifying the mood. These are one example, and theymay be taken as a feature from the result of analyzing the speech. Also,the variation of the feature over time may also be taken as a feature.Further, any other commonly known mood feature may also be used as thefeature.

Next, emotion recognition is performed using this feature. Firstly, foremotion recognition, learning is first performed using identificationanalysis, and an identification parameter coefficient is calculated fromthe feature of the previously disclosed speech data. These coefficientsare different for each emotion to be detected, and are the coefficients1-5 (1610-1613) in the emotion recognition database (FIG. 16). As aresult, for the 5 feature amounts X₁, X₂ . . . X₅, the formula for thedistinction coefficient is Z=a₁X₁+a₂X₂+ . . . +a₅X₅ from the 5coefficients a₁, a₂ . . . a₅ used for learning. This formula iscalculated for each emotion, and the smallest value is taken as theemotion. This technique is one example for specifying an emotion, butanother commonly known technique may also be used, for example atechnique such as neural networks or multivariate analysis.

Speaker recognition may use a process that is identical to emotionrecognition. For the coefficient of the identifying function, a speakerrecognition database (FIG. 15) is used. This technique is one examplefor identifying a speaker, but another commonly known technique may beused.

If a speaker could not be identified by emotion recognition, the speechmay actually be another sound. It is preferable to know what this othersound is, one example being environmental noise. For this judgment,environmental noises such as a buzzer, or music and the like, areidentified for the cutout frame. The identification technique may beidentical to that used for the emotion recognition/speaker recognition504. This technique is one example of identifying environmental noise inthe vicinity, but another commonly known technique may be used.

A speech database acquisition 505 is performed by the personal computer231, and outputs the result of the emotion recognition/speakerrecognition/peripheral noise recognition 504 to the speech database(FIG. 10). The information recorded is a person 1007 and an emotion 1008in the case of speech, and an environmental noise 1009 in the case ofnonspeech, for the corresponding frame. An end 506 is the end ofspeech/nonspeech activity detection.

The procedure of the sensor activity detection 303 will now be describedreferring to the flow chart of FIG. 6. This processing is performed bythe personal computer 231, and is the cutting out of a sensing signalfrom the sensor 211 in the same frame using the speech/nonspeech frameinformation output by the speech/nonspeech activity detection 302. Byperforming this processing, the sensing state of a person in the speechframe and nonspeech frame can be examined. A start 601 is the start ofsensor activity detection.

A sensor capture 602 is performed by the personal computer 231 whichcaptures a signal measured by a sensor, and captures the signal from thesensor 211. Also, assuming that this information is not only a signal,but also contains sensor-specific information, it is desirable to saveit as an ID-specific number, and preferable to store it in a customizedvalue database (FIG. 14) which manages data.

A sensor database acquisition 603 is performed by the personal computer231, and acquires data from the speech database to obtainspeech/nonspeech frames (FIG. 10).

A sensor cutout 604 is performed by the personal computer 231, andselects the starting time 1002 and closing time 1003 from data read bythe speech database read 603 to cut out a frame from the sensor signal.The sensor frame is then calculated using the starting time 1002 and theclosing time 1003. Finally, sensor signal cutout is performed based onthe result of activity detection, and saved in the memory of thepersonal computer 231.

A sensor database substitution 605 is performed by the personal computer231, and outputs the frame detected by the sensor cutout 604 to thesensor database (FIG. 11). The data to be recorded are the framestarting time 1102 and closing time 1103, and the sensor filename 1104cut out by the sensor activity detection 504. The frame starting time1102 and closing time 1103 are the time and date of the cutoff. When thefilename of the speech cut out by the sensor cutout 604 is decided, asensor signal cutout is performed from the result of the activitydetection by the sensor cutout 604 and stored in the memory, and this isconverted to a file for storing.

If data other than a sensor signal is saved by the sensor cutout 604, itis desirable to save it in the same way as a sensor signal. The filename1104 which is stored is preferably unique for easy search later. Thedetermined filename is then stored in the speech database. An end 606 isthe end of speech/sensor activity detection.

The processing of the frame-based sensor analysis 305 will now bedescribed referring to the flow chart of FIG. 7. This processing, whichis performed by the personal computer 231, analyzes the signal cut outby the sensor activity detection 304. By performing this processing, thesignal sensed from a person in the speech and nonspeech frames can beanalyzed. A start 701 is the start of the frame-based sensor analysis.

A sensor database acquisition 702 is performed by the personal computer231, and acquires data from the sensor database to obtain frame-basedsensor information (FIG. 11).

A feature extraction 703 is performed by the personal computer 231, andextracts the frame features from the frame-based sensor information. Thefeatures are the average, variance and standard deviation of the signalin the frame for each sensor. This procedure is an example of featureextraction, but another generally known procedure may also be used.

A sensor database substitution 704 is performed by the personal computer231, and outputs the features extracted by the feature extraction 703 tothe sensor database (FIG. 11). The information stored in the sensordatabase (FIG. 11) is an average 1106, variance 1107 and standarddeviation 1108 of the corresponding frame. An end 705 is the end of theframe-based sensor analysis.

The processing of the interest level judgment 306 will now be describedreferring to the flow chart of FIG. 8. This processing, which isperformed by the personal computer 231, determines the interest levelfrom the features for each frame analyzed by the frame-based soundanalysis 303 and frame-based sensor analysis 305. By performing thisprocessing, a signal sensed from the person in the speech and nonspeechframes can be analyzed, and the difference between the frames can befound.

In this processing, an interest level is calculated from the featurecorrelation in speech and nonspeech frames. An interest level for eachsensor and an interest level for each microphone, are also calculated.The reason for dividing the interest levels into two, is in order tofind which one of the participants is interested in the meeting from thesensor-based interest level, and to find which utterance was mostinteresting to the participants from the microphone-based interestlevel. A start 801 is the start of interest level judgment.

A sound database acquisition/sensor database acquisition 802 isperformed by the personal computer 231, and acquires data from the sounddatabase (FIG. 10) and sensor database (FIG. 11) to obtain frame-basedsound information and sensor information.

A sensor-based interest level extraction 803 is performed by thepersonal computer 231, and judges the interest level for each sensor inthe frame. A feature difference is found between speech and nonspeechframes for persons near the sensor, it being assumed that they have moreinterest in the meeting the larger this difference is. This is becausesome action is performed when there is an important utterance, and thedifference due to the action is large.

An interest level is calculated for a frame judged to be speech. Theinformation used for the analysis is the information in this frame, andthe information in the immediately preceding and immediately followingframes.

First, the recording is divided into speech and nonspeech for eachsensor, and normalization is performed.

The calculation formulae are features of normalized speech frames=speechframe features/(speech frame features+nonspeech frame features), andfeatures of normalized nonspeech frames=nonspeech frame features/(speechframe features+nonspeech frame features). The reason for performingnormalization is in order to lessen than the effect of scatteringbetween sensors by making the maximum value of the difference equal to1.

For example, in the case where sensor ID NO. 1 (1105) is used, thefeature (average) in a normalized speech frame is 3.2/(3.2+1.2)=0.73,the feature (average) in a normalized nonspeech frame is1.2/(3.2+1.2)=0.27, the feature (variance) in a normalized speech frameis 4.3/(4.3+3.1)=0.58, the feature (variance) in a normalized nonspeechframe is 3.1/(4.3+3.1)=0.42, the feature (standard deviation) in anormalized speech frame is 0.2/(0.2+0.8)=0.2, and the feature (standarddeviation) in a normalized nonspeech frame is 0.9/(0.2+0.8)=0.8.

Next, the interest level is calculated. The calculation formula is shownby Formula 1. A sensor coefficient is introduced to calculate acustomized interest level for a given person if the person detected bythe sensor can be identified. The range of values for the interest levelis 0-1. The closer the calculated value is to 1, the higher the interestlevel is. An interest level can be calculated for each sensor, and anyother procedure may be used.Sensor-based interest level=1/sensor average coefficient+sensor variancecoefficient+sensor standard deviation coefficient×(sensor averagecoefficient×(normalized speech frame feature(average)-normalizednonspeech frame feature(average))²+sensor variancecoefficient×(normalized speech frame feature(variance)−normalizednonspeech frame feature(variance))²+sensor averagecoefficient×(normalized speech frame feature(standarddistribution)−normalized nonspeech frame feature(standarddistribution))²)  Formula 1:

The sensor coefficient is normally 1, but if the person detected by thesensor can be identified, performance can be enhanced by using asuitable coefficient for the person from the correlation with thatperson. Specifically, it is preferable to use a coefficient (average)1410, coefficient (variance) 1411 and coefficient (standard deviation)1412 which are corresponding sensor ID coefficients in the customizedvalue database (FIG. 14). For example, in the case where sensor ID NO. 1(1105) is used, the interest level of sensor ID NO. 1 is given byFormula 2 using the coefficients for sensor ID NO. 1 (1407) in thecustomized value database (FIG. 14). This technique is one example ofspecifying the interest level from the sensor, but other techniquesknown in the art may also be used.0.6(0.73-0.27)²+1.0(0.58-0.42)²+0.4(0.2-0.8)²/0.6+1.0+0.4  Formula 2:

A microphone-based interest level extraction 804 is performed by thepersonal computer 231, and calculates the interest level for eachmicrophone in the frame. A feature difference between the framesimmediately preceding and immediately following the speech framerecorded by the microphone is calculated, and the interest level in anutterance is determined to be greater, the larger this difference is.

In the calculation, an average interest level is calculated for eachsensor found in the sensor-based interest level extraction 803, thisbeing the average for the corresponding microphone ID. The calculationformula is shown by Formula 3. This procedure is one example ofidentifying the interest level from the sensors, but other procedurescommonly known in the art may also be used.Microphone-based interest level=1/the number of sensors(interest levelof sensor 1+interest level of sensor 2+interest level of sensor3)  Formula 3:

An interest level database substitution 805 is processing performed bythe personal computer 231, the information calculated by thesensor-based interest level extraction being stored in the interestlevel database (sensor) (FIG. 12), and the information calculated by themicrophone-based interest level extraction being stored in the interestlevel database (microphone) (FIG. 13).

In the case of the interest level database (sensor) (FIG. 12), aninterest level is stored for each sensor in the frame. Also, whenpersonal information is included for each sensor by the sound databaseacquisition sensor database acquisition 802, this information isrecorded as person information 1206.

In the case of the interest level database (microphone) (FIG. 13), aninterest level is stored for the microphone detected in the frame. Whenspeech/nonspeech, person, emotion and environmental sound informationfor each microphone are included in the sound database acquisitionsensor database acquisition 802, this information is recorded as aspeech/nonspeech 1304, person 1306, emotion 1307 and environmental soundinformation 1308. An end 806 is the end of interest level judgment.

The processing of the display 307 will now be described referring to theflowchart of FIG. 9. In this processing, a screen is generated by thepersonal computer 231 using the interest level outputted by the interestlevel analysis 306. By performing such processing, user-friendliness canbe increased. In this processing, it is intended to create a more easilyunderstandable diagram by combining persons and times with the interestlevel. A start 901 is the start of the display.

An interest level database acquisition 902 is performed by the personalcomputer 231, and acquires data from an interest level database (sensor,microphone) (FIG. 12, FIG. 13).

A data processing 903 is processing performed by the personal computer231, and processes required information from data in the interest leveldatabase (sensor, microphone) (FIG. 12, FIG. 13). When processing isperformed, by first determining a time range and specifying a person, itcan be displayed at what times interest was shown, and in whoseutterances interest was shown.

To perform processing by time, it is necessary to specify a startingtime and a closing time. In the case of real time, several seconds afterthe present time are specified. To perform processing by person, it isnecessary to specify a person. Further, if useful data can be capturednot only from time and persons, but also from locations and team names,this may be used.

Processing is then performed to obtain the required information when thescreen is displayed. For example, FIG. 17 shows the change of interestlevel at each time for the participants in a meeting held in aconference room. This can be calculated from the database (sensor) ofFIG. 12.

For the calculation, A-E (1701-1705) consist of: 1. dividing thespecified time into still shorter time intervals, 2. calculating the sumof interest levels for persons included in the sensor ID, and 3.dividing by the total number of occasions to perform normalization. Byso doing, the interest level in a short time is calculated.

In the case of a total 1706, this is the sum of the interest level foreach user. In FIG. 17, although it is necessary to determine the axis ofa participant's interest level, in this patent, normalization isperformed and the range is 0-1. Therefore, assume that 0.5 which is themedian, is the value of the interest level axis. In this way, it can beshown how much interest the participants have in the meeting.

Further, FIG. 18, by displaying the interest level of the participants,shows the variation of interest level in a time series. This shows howmany useful utterances were made during the meeting. In FIG. 18, bycutting out only those parts with a high interest level, it can be shownhow many useful utterances were made. Further, it is also possible toplayback only speech in parts with a high interest level. This can becalculated from the interest level database (microphone) of FIG. 13.

The calculation of an interest level 1801 consists of 1. Furtherclassifying the specified time into short times, 2. Calculating the sumof interest levels included in the microphone ID in a short time, and 3.Dividing by the total number of occasions to perform normalization. Byso doing, the variation of interest level in a meeting can be displayed,and it can be shown how long a meeting with useful utterances tookplace. The closer the value is to 1, the higher the interest level is.Further, in the color specification 1802, a darker color is selected,the closer to 1 the interest level is.

The meeting viewer 241 in the interest level analysis image of FIG. 2,shows which participants made useful utterances. Participants A-E arepersons, the circles show utterances with a high interest level, and thelines joining circles show the person who spoke next. It is seen thatthe larger the circle, the more useful utterances there are, and thethicker the line, the larger the number of occasions when there werefollowing utterances.

This calculation can be performed from the interest level database(microphone) of FIG. 13. In the calculation, the circles are the sum ofinterest levels included in the microphone. ID in a specified time, andthe lines show the sequence of utterances by persons immediately beforeand after the utterance frame. In this way, it can be shown who capturedmost people's attention, and made useful utterances. An end 905 is theend of the display.

In the speech/nonspeech activity detection 302, speech/nonspeechanalysis is performed from the sound, and the output data at that timeis preferably managed as a database referred to as a speech database.FIG. 10 shows one example of a speech database.

The structure of the speech database of FIG. 10 is shown below. The ID(1001) is an ID denoting a unique number. This preferably refers to thesame frames and same ID as the database (FIG. 11). The starting time1002 is the starting time in a frame output from the speech/nonspeechactivity detection 302. The closing time 1003 is the closing time in aframe output from the speech/nonspeech activity detection 302. Thestarting time 1002 and closing time 903 are stored together with thedate and time. The microphone ID 1004 is the unique ID of the microphoneused for sound recording. The speech/nonspeech 1005 stores the resultdetermined in the speech/nonspeech activity detection 302. The savedfile 1006 is the result of cutting out the sound based on the framedetermined by the speech/nonspeech activity detection 302 and storingthis as a file, and a filename is stored for the purpose of easyreference later. The person 1007 stores the result of the speakerrecognition performed by the emotion recognition/speaker recognition504. The emotion 1008 stores the result of the emotion recognitionperformed by the emotion recognition/speaker recognition 504. Also, theenvironmental noise 1009 stores the identification result of theenvironmental sound recognition 505.

In the sensor activity detection 304, when sensor signal activitydetection is performed using frames detected by the speech/nonspeechactivity detection 302, the output data is preferably managed as adatabase. FIG. 11 shows one example of a sensor database.

The structure of the sensor database of FIG. 11 is shown below. The ID(1101) is an ID which shows an unique number. It is preferable that thisis the same frame and ID as the speech database (FIG. 10). The startingtime 1102 is the starting time in a frame output from the sensoractivity detection 304. The closing time 1103 is the closing time in aframe output from the sensor activity detection 304. The starting time1102 and closing time 1103 are stored as the date and time. The savedfile 1104 is the result of cutting out the signal based on the framedetermined by the sensor activity detection 304, and storing this as afile, and a filename is stored for the purpose of easy reference later.The sensor ID (1105) is the unique ID of the sensor used for sensing.The average 1006 stores the average in the frames for which the featureextraction 703 was performed. The variance 1107 stores the variance inthe frames for which the feature extraction 703 was performed. Thestandard deviation 1008 stores the standard deviation in the frames forwhich the feature extraction 703 was performed.

When calculating the interest level, it is preferable to manage anoutput database, which is referred to as an interest level database. Theinterest level database is preferably calculated for eachmicrophone/sensor, and FIG. 12 shows an example of the interest leveldatabase for each sensor.

The structure of the interest level database for each sensor in FIG. 12is shown below. An ID (1201) is an ID showing a unique number. In thecase of the same frame, it is preferable that the ID (1201) is the sameID as an ID (1301) of the interest level database (FIG. 13), the ID(1001) of the speech database (FIG. 10), and the ID (1101) of the sensordatabase (FIG. 11). A starting time 1202 stores the starting timecalculated by the sensor-based interest level extraction 803. A closingtime 1203 stores the end time calculated by the sensor-based interestlevel extraction 803. A speech/nonspeech 1204 stores the analysis resultcalculated by the sensor-based interest level extraction 803. A sensorID NO. 1 (1205) stores the analysis result of the sensor for which thesensor ID is NO. 1. Examples of this value are a person 1206 andinterest level 1207, which store the person and interest levelcalculated by the sensor-based interest level extraction 803.

FIG. 13 shows one example of the interest level database for eachmicrophone. The structure of the interest level database for eachmicrophone in FIG. 13 is shown below. The ID (1301) is an ID which showsa unique number. For the same frame, it is preferable that the ID (1301)is the same ID as the ID (1201) of the interest level database (FIG.12), the ID (1001) of the speech database (FIG. 10), and the ID (1101)of the sensor database (FIG. 11). A starting time 1302 stores thestarting time calculated by the microphone-based interest levelextraction 804. A closing time 1303 stores the closing time calculatedby the microphone-based interest level extraction 804. Aspeech/nonspeech 1304 stores the analysis result calculated by themicrophone-based interest level extraction 804. A microphone ID NO. 1(1305) stores the analysis result of the sound for which the microphoneID is NO. 1. Examples of this value are a person 1306, emotion 1307,interest level 1308, an environmental sound 1309, and these store theperson, emotion, interest level and environmental sound calculated bythe microphone-based interest level extraction 804.

In the speech/nonspeech activity detection 302 or the interest leveljudgment 306, sound and sensor signal analyses are performed, and toincrease the precision of these analyses, information pertinent to theanalyzed person is preferably added. For this purpose, if the personusing a microphone or sensor is known, a database containing informationspecific to this person is preferably used. The database which storespersonal characteristics is referred to as a customized value database,and FIG. 14 shows an example of this database.

An ID (1401) stores the names of the microphone ID and sensor ID. In thecase of the microphone ID, it may be for example microphone ID No.1(1402), and in the case of the sensor ID, it may be for example sensorID No. 1 (1407). For the microphone ID NO. 1 (1402), if the microphoneis installed, an installation location 1403 is stored, if only oneperson uses it, a person 1404 is stored, and if threshold values forcustomizing the location and the person are used, values are stored in athreshold value (energy) 1405 and threshold value (zero cross) 1406. Thesituation is identical for the sensor ID NO. 1 (1407). If the sensor isinstalled, an installation location 1408 is stored, if only one personuses it, a person 1409 is stored, and if a coefficient is used forcustomizing the location and person, values are stored in a coefficient(average) 1410, coefficient (variance) 1411, and a coefficient (standarddeviation) 1412.

The frame-based analysis 303 is processing to analyze a sound cut out bythe speech/nonspeech activity detection 302. In particular, to grasp thestate of a person from speech, a database containing coefficients andfeature amounts representing the state is required, and this ispreferably managed. A database containing coefficients and features forspeaker recognition is referred to as a speaker recognition database,and a database containing coefficients and features for emotionrecognition is referred to as an emotion recognition database. FIG. 15shows an example of a speaker recognition database, and FIG. 16 shows anexample of an emotion recognition database.

First, one example (FIG. 15) of a speaker recognition database will bedescribed. An item 1501 is an identifying item, and this item identifiesmale/female (male 1502, female 1505), or identifies a person (TaroYamada 1506, Hanako Yamada 1507). The information is not limited tothis, and may also include for example age or the like when it isdesired to identify this. The values contained in the item may beclassified into standard values 1503 and customized values 1504. Thestandard values 1503 are general values and are recorded beforehand bythe personal computer 231. The customized values 1504 are values adaptedto the individual, are transmitted together with the sensor signal fromthe sensor 211, and are stored in the speaker recognition database (FIG.15). Further, each item consists of several coefficients, and in thecase of the speaker recognition database (FIG. 15), these are denoted bythe coefficients 1-5 (1508-1502).

Next, FIG. 16 will be described. An item 1601 is an identifying item,and this item identifies male/female (male 1602, female 1608), oridentifies a person (Taro Yamada 1609). The item 1601 may show aperson's emotion, and it may show emotion according to age as in thecase of the item 1501 of FIG. 15. The values contained in this item maybe for example emotions (anger 1602, neutrality 1605, laughter 1606,sadness 1607). The values are not limited to these, and other emotionsmay also be used when it is desired to identify them. The values may beclassified into standard values 1603 and customized values 1604. Thecustomized values 1604 are values adapted to the individual, aretransmitted together with the sensor signal from the sensor 211, and arestored in the emotion recognition database (FIG. 16). Further, each itemconsists of several coefficients, and in the case of the emotionrecognition database (FIG. 16), these are denoted by the coefficients1-5 (1609-1612).

As described above, in the embodiments, by finding correlations frommicrophone and sensor signals, an analysis is performed as to how muchinterest the participants have in the meeting. By displaying thisresult, the activity of the participants in the meeting can be evaluatedand the state of the meeting can be evaluated for persons who are notpresent, and by saving this information, it can be used for future loganalysis.

Here, the sound captured by a microphone was used as a signal forcalculating frames, but if it can be used for calculating frames,another signal such as an image captured by a camera may also be used.

Further, in the embodiments, if a signal can be captured by a sensor, itcan be used for analysis, so other sensors may be used such as a gravitysensor, acceleration sensor, pH and a conductivity sensor, RFID sensor,gas sensor, torque sensor, microsensor, motion sensor, laser sensor,pressure sensor, location sensor, liquid and bulk level sensor,temperature sensor, temperature sensor, thermistor, climate sensor,proximity sensor, gradient sensor, photosensor, optical sensor,photovoltaic sensor, oxygen sensor, ultraviolet radiation sensor,magnetometric sensor, humidity sensor, color sensor, vibration sensor,infrared sensor, electric current and voltage sensor, or flow ratesensor or the like.

What is claimed is:
 1. A conversational speech analyzing system,comprising: a data capturing unit configured to capture data of aconversation or meeting; a motion sensor configured to capture motioninformation of a person involved in the conversation or meeting; and acomputer, including: a frame detection unit configured to detect framesbased on the captured data, a frame being a time segment of theconversation or meeting, and wherein the frame detection unit isconfigured to detect a first set of frames of data associated with afirst person by using a second set of frames of data associated with asecond person, a feature extraction unit configured to extract featureinformation from the sensor information of the first person in the firstset of frames, a calculation unit configured to calculate a state of thefirst person in the conversation or meeting, including an interest levelof the first person in the meeting, and to calculate a change of thestate of the first person, including a change in interest level of thefirst person in the meeting, and a display unit configured to display atleast one of: the state of the first person involved in the conversationor meeting, including a symbol of the first person, the symbol includinga circle whose radius corresponds to a quantity of comments made by thefirst person in the conversation or meeting found to be useful orinteresting by other participants, a plurality of time segments forminga time series, each time segment represented by a symbol including arectangle whose shade or color corresponds to an aggregate of interestlevels of persons involved in the conversation or meeting, and changesof the state of the first person shown in chronological order, includinga graph that corresponds to the change over time in an interest level ofthe first person in the conversation or meeting.
 2. The conversationalspeech analyzing system according to claim 1, wherein the sensorincludes an acceleration sensor.
 3. The conversational speech analyzingsystem according to claim 1, wherein the data capturing unit configuredto capture data includes a microphone configured to capture sound data.4. The conversational speech analyzing system according to claim 1,wherein the display unit is configured to display the state of the firstperson.
 5. The conversational speech analyzing system according to claim4, wherein the display unit is configured to display the state of thefirst person involved in the conversation or meeting, including a symbolof the first person, the symbol including a circle whose radiuscorresponds to a quantity of comments made by the first person in theconversation or meeting found to be useful or interesting by otherparticipants.
 6. The conversational speech analyzing system according toclaim 1, wherein the display unit is configured to display, inchronological order, changes of the state of the first person shown inchronological order, including a graph that corresponds to the changeover time in an interest level of the first person in the conversationor meeting.
 7. The conversational speech analyzing system according toclaim 1, wherein the display unit is configured to display a pluralityof time segments forming a time series, each time segment represented bya symbol including a rectangle whose shade or color corresponds to anaggregate of interest levels of persons involved in the conversation ormeeting.
 8. A conversational speech analyzing system, comprising: a datacapturing unit configured to capture data of a conversation or meeting;a motion sensor configured to capture motion information of a personinvolved in the conversation or meeting; and a computer, including: aframe detection unit configured to detect frames based on the captureddata, a frame being a time segment of the conversation or meeting, andwherein the frame detection unit is configured to detect a first set offrames of data associated with a first person by using a second set offrames of data associated with a second person, a feature extractionunit configured to extract, for the first person involved in theconversation or meeting, feature information from the sensor informationin the first set of frames, an interest level extraction unit configuredto extract an interest level, within a particular frame in theconversation, for the first person involved in the conversation ormeeting, and a display unit configured to display at least one of: asymbol of the first person involved in the conversation or meeting, thesymbol including a circle whose radius corresponds to a quantity ofcomments made by the first person in the conversation or meeting foundto be useful or interesting by other participants, a plurality of timesegments forming a time series, each time segment represented by asymbol including a rectangle whose shade or color corresponds to anaggregate of interest levels of persons involved in the conversation ormeeting, and a graph that corresponds to the change over time in aninterest level of the first person in the conversation or meeting. 9.The conversational speech analyzing system according to claim 8, whereinthe data capturing unit configured to capture data includes a microphoneconfigured to capture sound data.
 10. The conversational speechanalyzing system according to claim 8, wherein the display unit isconfigured to display a symbol of the first person involved in theconversation or meeting, the symbol including a circle whose radiuscorresponds to a quantity of comments made by the first person in theconversation or meeting found to be useful or interesting by otherparticipants.
 11. The conversational speech analyzing system accordingto claim 8, wherein the display unit is configured to display a graphthat corresponds to the change over time in an interest level of thefirst person in the conversation or meeting.
 12. The conversationalspeech analyzing system according to claim 8, wherein the display unitis configured to display a plurality of time segments forming a timeseries, each time segment represented by a symbol including a rectanglewhose shade or color corresponds to an aggregate of interest levels ofpersons involved in the conversation or meeting.
 13. A conversationalspeech analyzing system, comprising: a data capturing means forcapturing data of a meeting; a motion sensing means for capturing motioninformation of persons involved in the meeting; and a computing means,including: a frame detection means for detecting frames based on thecaptured data, a frame being a time segment of the meeting, and whereinthe frame detection means detects a first set of frames of dataassociated with a first person by using a second set of frames of dataassociated with a second person, a feature extraction means forextracting feature information from the sensor information of the firstperson in the first set of frames, a calculation means for calculating astate of the first person in the conversation or meeting, including aninterest level of the first person in the meeting, and for calculating achange of the state of the first person, including a change in interestlevel of the first person in the meeting, and a display means fordisplaying at least one of: the state of the first person involved inthe conversation or meeting, including a symbol of the first person, thesymbol including a circle whose radius corresponds to a quantity ofcomments made by the first person in the conversation or meeting foundto be useful or interesting by other participants, a plurality of timesegments forming a time series, each time segment represented by asymbol including a rectangle whose shade or color corresponds to anaggregate of interest levels of persons involved in the conversation ormeeting, and changes of the state of the first person shown inchronological order, including a graph that corresponds to the changeover time in an interest level of the first person in the conversationor meeting.
 14. The conversational speech analyzing system according toclaim 13, wherein the display means is for displaying the state of thefirst person.
 15. The conversational speech analyzing system accordingto claim 13, wherein the display means includes a symbol of the firstperson involved in the conversation or meeting, the symbol including acircle whose radius corresponds to a quantity of comments made by thefirst person in the conversation or meeting found to be useful orinteresting by other participants.
 16. The conversational speechanalyzing system according to claim 13, wherein the display means is fordisplaying, in chronological order, changes of the state of the firstperson, including a graph that corresponds to the change over time in aninterest level of the first person in the conversation or meeting. 17.The conversational speech analyzing system according to claim 13,wherein the display means is for displaying a plurality of time segmentsforming a time series, each time segment represented by a symbolincluding a rectangle whose shade or color corresponds to an aggregateof interest levels of persons involved in the conversation or meeting.18. The conversational speech analyzing system according to claim 13,wherein the sensing means includes an acceleration sensing means. 19.The conversational speech analyzing system according to claim 13,wherein the data means for capturing data includes a microphone forcapturing sound data.